Combined power and cooling rack supporting an electronics rack(S)

ABSTRACT

A combined power and cooling apparatus is provided facilitating powering and cooling one or more electronics racks, which are distinct from the power and cooling apparatus. The power and cooling apparatus includes a frame, one or more bulk power assemblies associated with the frame, and one or more heat exchange assemblies associated with the frame. The one or more bulk power assemblies are configured to provide power to the one or more electronics racks, and the one or more heat exchange assemblies are configured to cool system coolant provided to the one or more electronics racks. Heat is transferred by the one or more heat exchange assemblies from the system coolant to a facility coolant. In operation, the power and cooling apparatus is coupled to provide both power and cooling to the one or more electronics racks.

BACKGROUND

The present invention relates in general to apparatuses and methods forfacilitating operation of liquid-cooled, rack-mounted assemblages ofindividual electronics units, such as rack-mounted computer serverunits.

The power dissipation of integrated circuit chips, and the modulescontaining the chips, continues to increase in order to achieveincreases in processor performance. This trend poses a cooling challengeat both the module and system levels. Increased airflow rates are neededto effectively cool high power modules and to limit the temperature ofthe air that is exhausted into the computer center.

In many large server applications, processors along with theirassociated electronics (e.g., memory, disk drives, power supplies, etc.)are packaged in removable drawer configurations stacked within a rack orframe. In other cases, the electronics may be in fixed locations withinthe rack or frame. Typically, the components are cooled by air moving inparallel airflow paths, usually front-to-back, impelled by one or moreair moving devices (e.g., fans or blowers). In some cases it may bepossible to handle increased power dissipation within a single drawer byproviding greater airflow, through the use of a more powerful air movingdevice or by increasing the rotational speed (i.e., RPMs) of an existingair moving device. However, this approach is becoming problematic at therack level in the context of a computer installation (i.e., datacenter).

The sensible heat load carried by the air exiting the rack is stressingthe availability of the room air-conditioning to effectively handle theload. This is especially true for large installations with “serverfarms” or large banks of computer racks close together. In suchinstallations, liquid cooling (e.g., water cooling) is an attractivetechnology to manage the higher heat fluxes. The liquid absorbs the heatdissipated by the components/modules in an efficient manner. Typically,the heat is ultimately transferred from the liquid to an outsideenvironment, whether air or other liquid coolants.

BRIEF SUMMARY

In one aspect, provided herein is an apparatus which includes: a frame;at least one bulk power assembly; and at least one heat exchangeassembly. The at least one bulk power assembly is associated with theframe, and is configured to provide power to at least one electronicsrack, wherein the apparatus is distinct from the at least oneelectronics rack. The at least one heat exchange assembly is associatedwith the frame, and is configured to cool system coolant provided to theat least one electronics rack to facilitate cooling of the at least oneelectronics rack, wherein heat is transferred by the at least one heatexchange assembly from the system coolant to a facility coolant, andwherein the apparatus provides both power and cooling to the at leastone electronics rack.

In another aspect, a data center is provided which includes at least oneelectronics rack comprising at least one electronic subsystem to bepowered and cooled, and a power and cooling apparatus for powering andcooling the at least one electronics rack. The power and coolingapparatus is distinct from the at least one electronics rack, andincludes: a frame; at least one bulk power assembly associated with theframe and configured to provide power to the at least one electronicsrack; and at least one heat exchange assembly associated with the frameand configured to cool system coolant provided to the at least oneelectronics rack, wherein heat is transferred by the at least one heatexchange assembly from the system coolant to a facility coolant, andwherein the power and cooling apparatus provides both power and coolingto the at least one electronics rack.

In a further aspect, a method of providing an apparatus for facilitatingpowering and cooling of at least one electronics rack is provided. Themethod includes: providing a frame; associating at least one bulk powerassembly with the frame, the at least one bulk power assembly beingconfigured to provide power to the at least one electronics rack,wherein the frame with the associated at least one bulk power assemblyare distinct from the at least one electronics rack; and associating atleast one heat exchange assembly with the frame, the at least one heatexchange assembly being configured to cool system coolant provided tothe at least one electronics rack, wherein heat is transferred by the atleast one heat exchange assembly from the system coolant to a facilitycoolant, and wherein the frame with the associated at least one bulkpower assembly and associated at least one heat exchange assemblyprovides both power and cooling to the at least one electronics rack.

Additional features and advantages are realized through the techniquesof the present invention. Other embodiments and aspects of the inventionare described in detail herein and are considered a part of the claimedinvention.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

One or more aspects of the present invention are particularly pointedout and distinctly claimed as examples in the claims at the conclusionof the specification. The foregoing and other objects, features, andadvantages of the invention are apparent from the following detaileddescription taken in conjunction with the accompanying drawings inwhich:

FIG. 1 depicts one embodiment of a conventional raised floor layout ofan air-cooled data center;

FIG. 2 is a front elevational view of one embodiment of an at leastpartially liquid-cooled electronics rack comprising multiple electronicsubsystems, in accordance with one or more aspects of the presentinvention;

FIG. 3 is a schematic of one embodiment of an electronics rack, whereinan electronic module (or component) is liquid-cooled by system coolant(provided by one or more coolant conditioning units disposed within theelectronics rack) passing through a liquid-cooled structure coupled tothe electronic module, in accordance with one or more aspects of thepresent invention;

FIG. 4 is a schematic of one embodiment of a coolant conditioning unitdisposed within a liquid-cooled electronics rack, in accordance with oneor more aspects of the present invention;

FIG. 5 is a plan view of one embodiment of an electronic subsystemlayout illustrating an air and liquid cooling system for coolingcomponents of the electronic subsystem, in accordance with one or moreaspects of the present invention;

FIG. 6 depicts one detailed embodiment of a partially-assembledelectronic subsystem layout, wherein the electronic subsystem includeseight heat-generating electronics components to be actively cooled, eachhaving a respective liquid-cooled structure of a liquid-based coolingsystem coupled thereto, in accordance with one or more aspects of thepresent invention;

FIG. 7 depicts one embodiment of a data center comprising multipleelectronics racks powered and cooled by a power and cooling rack orapparatus, in accordance with one or more aspects of the presentinvention;

FIG. 8A depicts one embodiment of a power and cooling rack, inaccordance with one or more aspects of the present invention;

FIG. 8B depicts one embodiment of a modular cooling unit for the powerand cooling rack of FIG. 8A, in accordance with one or more aspects ofthe present invention;

FIG. 8C is a flowchart of one process embodiment for controllingoperation of the system coolant pump of the modular cooling unit of FIG.8B, in accordance with one or more aspects of the present invention;

FIG. 8D depicts one embodiment of a heat exchange assembly for the powerand cooling rack of FIG. 8A, in accordance with one or more aspects ofthe present invention;

FIG. 8E is a flowchart of one process embodiment for controlling flow offacility coolant through the heat exchange assembly of FIG. 8D, inaccordance with one or more aspects of the present invention; and

FIG. 8F is a plan view of one embodiment of a facility coolant-cooledcold plate coupled to a bulk power assembly of the power and coolingrack of FIG. 8A, in accordance with one or more aspects of the presentinvention.

DETAILED DESCRIPTION

As used herein, the terms “electronics rack”, and “rack-mountedelectronic equipment” are used interchangeably, and unless otherwisespecified include any housing, frame, rack, compartment, blade serversystem, etc., having one or more heat-generating components of acomputer system or electronics system, and may be, for example, astand-alone computer processor having high, mid or low end processingcapability. In one embodiment, an electronics rack may comprise multipleelectronic subsystems, each having one or more heat-generatingcomponents disposed therein requiring cooling. “Electronic subsystem”refers to any sub-housing, blade, book, drawer, node, compartment, etc.,of an electronics rack having one or more heat-generating electroniccomponents disposed therein. Each electronic subsystem of an electronicsrack may be movable or fixed relative to the electronics rack, with therack-mounted electronics drawers of a multi-drawer rack unit and bladesof a blade center system being two examples of subsystems of anelectronics rack to be cooled. In one specific example, each electronicsubsystem comprises a server node of a multi-server rack.

“Electronic component” refers to any heat-generating electroniccomponent of, for example, a computer system or other electronics unitrequiring cooling. By way of example, an electronic component maycomprise one or more integrated circuit dies and/or other electronicdevices to be cooled, including one or more processor dies, memory diesand memory support dies. As a further example, an electronic componentmay comprise one or more bare dies or one or more packaged dies disposedon a common carrier. As used herein, “primary heat-generating component”refers to a primary heat-generating electronic component within anelectronic subsystem, while “secondary heat-generating component” refersto an electronic component of the electronic subsystem generating lessheat than the primary heat-generating component to be cooled. Further,unless otherwise specified herein, the term “liquid-cooled cold plate”refers to any conventional thermally conductive structure having one ormore channels or passageways formed therein for flowing of liquidcoolant therethrough. In addition, “metallurgically bonded” refersgenerally herein to two components being welded, brazed or solderedtogether by any means.

As used herein, “liquid-to-air heat exchanger” means any heat exchangemechanism characterized as described herein through which liquid coolantcan circulate; and includes, one or more discrete liquid-to-air heatexchange mechanisms coupled either in series or in parallel. Aliquid-to-air heat exchanger may comprise, for example, one or morecoolant flow paths, formed of thermally conductive tubing (such ascopper or other tubing) in thermal or mechanical contact with aplurality of air-cooled cooling fins. Size, configuration andconstruction of the liquid-to-air heat exchanger can vary withoutdeparting from the scope of the invention disclosed herein. A“liquid-to-liquid heat exchanger” may comprise, for example, two or morecoolant flow paths, formed of thermally conductive tubing (such ascopper or other tubing) in thermal communication with each other. Size,configuration and construction of the liquid-to-liquid heat exchangercan vary without departing from the scope of the invention disclosedherein. Further, “data center” refers to a computer installationcontaining one or more electronics racks to be cooled. As a specificexample, a data center may include one or more rows of rack-mountedcomputing units, such as server units.

One example of a liquid coolant (for example, the facility or systemcoolants discussed below) is water. However, the concepts disclosedherein are readily adapted to use with other types of coolant. Forexample, one or more of the coolants may comprise a brine, afluorocarbon liquid, a liquid metal, or other similar coolant, orrefrigerant, while still maintaining the advantages and unique featuresof the present invention.

Reference is made below to the drawings (which are not drawn to scale tofacilitate understanding of the invention), wherein the same referencenumbers used throughout different figures designate the same or similarcomponents.

FIG. 1 depicts a raised floor layout of an air cooled data center 100typical in the prior art, wherein multiple electronics racks 110 aredisposed in one or more rows. A data center such as depicted in FIG. 1may house several hundred, or even several thousand microprocessors. Inthe arrangement illustrated, cooled air enters the data center viaperforated floor tiles 160 from a cool air plenum 145 defined betweenthe raised floor 140 and a base or sub-floor 165 of the room. Cooled airis taken in through louvered covers at air inlet sides 120 of theelectronics racks and expelled through the back (i.e., air outlet sides130) of the electronics racks. Each electronics rack 110 may have one ormore air moving devices (e.g., fans or blowers) to provide forcedinlet-to-outlet airflow to cool the electronic devices within thesubsystem(s) of the rack. The supply air plenum 145 provides conditionedand cooled air to the air-inlet sides of the electronics racks viaperforated floor tiles 160 disposed in a “cold” aisle of the datacenter. The conditioned and cooled air is supplied to plenum 145 by oneor more air conditioning units 150, also disposed within the data center100. Room air is taken into each air conditioning unit 150 near an upperportion thereof. This room air may comprise in part exhausted air fromthe “hot” aisles of the computer installation defined, for example, byopposing air outlet sides 130 of electronics racks 110.

Due to the ever-increasing airflow requirements through electronicsracks, and the limits of air distribution within the typical data centerinstallation, liquid-assisted cooling is being combined with theconventional air-cooling. FIGS. 2-6 illustrate one embodiment of a datacenter implementation employing a liquid-assisted cooling system withone or more cold plates coupled to high heat-generating electroniccomponents disposed within the electronics racks.

FIG. 2 depicts one embodiment of a partially liquid-cooled electronicsrack 200. As illustrated, liquid-cooled electronics rack 200 comprises aplurality of electronic subsystems 210, which may be processor or servernodes. A bulk power regulator 220 is shown disposed at an upper portionof liquid-cooled electronics rack 200, and two coolant conditioningunits (CCUs) 230 are disposed in a lower portion of the liquid-cooledelectronics rack. In the embodiments described herein, the coolant isassumed to be water or an aqueous-based solution (by way of exampleonly).

In addition to CCUs 230, the cooling system includes a system watersupply manifold 231, a system water return manifold 232, andmanifold-to-node fluid connect hoses 233 coupling system water supplymanifold 231 to electronic subsystems 210, and node-to-manifold fluidconnect hoses 234 coupling the individual electronic subsystems 210 tosystem water return manifold 232. Each CCU 230 is in fluid communicationwith system water supply manifold 231 via a respective system watersupply hose 235, and each CCU 230 is in fluid communication with systemwater return manifold 232 via a respective system water return hose 236.

As illustrated, a portion of the heat load of the electronic subsystemsis transferred from the system water to cooler facility water suppliedby facility water supply line 240 and facility water return line 241disposed, in the illustrated embodiment, in the space between a raisedfloor 201 and a base floor 202.

FIG. 3 schematically illustrates operation of the cooling system of FIG.2, wherein a liquid-cooled cold plate 300 is shown coupled to anelectronic module 301 of an electronic subsystem 210 within theliquid-cooled electronics rack 200. Heat is removed from electronicmodule 301 via the system coolant circulated via pump 320 through coldplate 300 within the system coolant loop defined by liquid-to-liquidheat exchanger 321 of coolant conditioning unit 230, lines 322, 323 andcold plate 300. The system coolant loop and coolant conditioning unitare designed to provide coolant of a controlled temperature andpressure, as well as controlled chemistry and cleanliness to theelectronic module(s). Furthermore, the system coolant is physicallyseparate from the less controlled facility coolant in lines 240, 241, towhich heat is ultimately transferred.

FIG. 4 depicts a more detailed embodiment of a coolant conditioning unit230. As shown in FIG. 4, coolant conditioning unit 230 includes a firstcoolant loop wherein chilled, facility coolant is supplied 410 andpasses through a control valve 420 driven by a motor 425. Valve 420determines an amount of facility coolant to be passed through heatexchanger 321, with a portion of the facility coolant possibly beingreturned directly via a bypass orifice 435. The coolant conditioningunit further includes a second coolant loop with a reservoir tank 440from which system coolant is pumped, either by pump 450 or pump 451,into the heat exchanger 321 for conditioning and output thereof, ascooled system coolant to the electronics rack to be cooled. The cooledsystem coolant is supplied to the system water supply manifold andsystem water return manifold of the liquid-cooled electronics rack viathe system water supply hose 235 and system water return hose 236.

FIG. 5 depicts one embodiment of an electronic subsystem 513 componentlayout wherein one or more air moving devices 511 provide forced airflow 515 to cool multiple components 512 within electronic subsystem513. Cool air is taken in through a front 531 and exhausted out a back533 of the drawer. The multiple components to be cooled include multipleprocessor modules to which liquid-cooled cold plates 520 (of aliquid-based cooling system) are coupled, as well as multiple arrays ofmemory modules 530 (e.g., dual in-line memory modules (DIMMs)) andmultiple rows of memory support modules 532 (e.g., DIMM control modules)to which air-cooled heat sinks are coupled. In the embodimentillustrated, memory modules 530 and the memory support modules 532 arepartially arrayed near front 531 of electronic subsystem 513, andpartially arrayed near back 533 of electronic subsystem 513. Also, inthe embodiment of FIG. 5, memory modules 530 and the memory supportmodules 532 are cooled by air flow 515 across the electronic subsystem.

The illustrated liquid-based cooling system further includes multiplecoolant-carrying tubes connected to and in fluid communication withliquid-cooled cold plates 520. The coolant-carrying tubes comprise setsof coolant-carrying tubes, with each set including (for example) acoolant supply tube 540, a bridge tube 541 and a coolant return tube542. In this example, each set of tubes provides liquid coolant to aseries-connected pair of cold plates 520 (coupled to a pair of processormodules). Coolant flows into a first cold plate of each pair via thecoolant supply tube 540 and from the first cold plate to a second coldplate of the pair via bridge tube or line 541, which may or may not bethermally conductive. From the second cold plate of the pair, coolant isreturned through the respective coolant return tube 542.

FIG. 6 depicts in greater detail an alternate electronics drawer layoutcomprising eight processor modules, each having a respectiveliquid-cooled cold plate of a liquid-based cooling system coupledthereto. The liquid-based cooling system is shown to further includeassociated coolant-carrying tubes for facilitating passage of liquidcoolant through the liquid-cooled cold plates and a header subassemblyto facilitate distribution of liquid coolant to and return of liquidcoolant from the liquid-cooled cold plates. By way of specific example,the liquid coolant passing through the liquid-based cooling subsystem ischilled water.

As noted, various liquid coolants significantly outperform air in thetask of removing heat from heat-generating electronic components of anelectronic system, and thereby more effectively maintain the componentsat a desirable temperature for enhanced reliability and peakperformance. As liquid-based cooling systems are designed and deployed,it is advantageous to architect systems which maximize reliability andminimize the potential for leaks while meeting all other mechanical,electrical and chemical requirements of a given electronic systemimplementation. These more robust cooling systems have unique problemsin their assembly and implementation. For example, one assembly solutionis to utilize multiple fittings within the electronic system, and useflexible plastic or rubber tubing to connect headers, cold plates, pumpsand other components. However, such a solution may not meet a givencustomer's specifications.

Thus, presented herein in one aspect is a robust liquid-assisted coolingsystem specially preconfigured and prefabricated as a monolithicstructure for positioning within a particular electronics drawer.

FIG. 6 is an isometric view of one embodiment of an electronics drawerand monolithic cooling system, in accordance with an aspect of thepresent invention. The depicted planar server assembly includes amulti-layer printed circuit board to which memory DIMM sockets andvarious electronic components to be cooled are attached both physicallyand electrically. In the cooling system depicted, a supply header isprovided to distribute liquid coolant from a single inlet to multipleparallel coolant flow paths and a return header collects exhaustedcoolant from the multiple parallel coolant flow paths into a singleoutlet. Each parallel coolant flow path includes one or more cold platesin series flow arrangement to cool one or more electronic components towhich the cold plates are mechanically and thermally coupled. The numberof parallel paths and the number of series-connected liquid-cooled coldplates depends, for example on the desired device temperature, availablecoolant temperature and coolant flow rate, and the total heat load beingdissipated from each electronic component.

More particularly, FIG. 6 depicts a partially assembled electronicsystem 613 and an assembled liquid-based cooling system 615 coupled toprimary heat-generating components (e.g., including processor dies) tobe cooled. In this embodiment, the electronics system is configured for(or as) an electronics drawer of an electronics rack, and includes, byway of example, a support substrate or planar board 605, a plurality ofmemory module sockets 610 (with the memory modules (e.g., dual in-linememory modules) not shown), multiple rows of memory support modules 632(each having coupled thereto an air-cooled heat sink 634), and multipleprocessor modules (not shown) disposed below the liquid-cooled coldplates 620 of the liquid-based cooling system 615.

In addition to liquid-cooled cold plates 620, liquid-based coolingsystem 615 includes multiple coolant-carrying tubes, including coolantsupply tubes 640 and coolant return tubes 642 in fluid communicationwith respective liquid-cooled cold plates 620. The coolant-carryingtubes 640, 642 are also connected to a header (or manifold) subassembly650 which facilitates distribution of liquid coolant to the coolantsupply tubes and return of liquid coolant from the coolant return tubes642. In this embodiment, the air-cooled heat sinks 634 coupled to memorysupport modules 632 closer to front 631 of electronics drawer 613 areshorter in height than the air-cooled heat sinks 634′ coupled to memorysupport modules 632 near back 633 of electronics drawer 613. This sizedifference is to accommodate the coolant-carrying tubes 640, 642 since,in this embodiment, the header subassembly 650 is at the front 631 ofthe electronics drawer and the multiple liquid-cooled cold plates 620are in the middle of the drawer.

Liquid-based cooling system 615 comprises a pre-configured monolithicstructure which includes multiple (pre-assembled) liquid-cooled coldplates 620 configured and disposed in spaced relation to engagerespective heat-generating electronic components. Each liquid-cooledcold plate 620 includes, in this embodiment, a liquid coolant inlet anda liquid coolant outlet, as well as an attachment subassembly (i.e., acold plate/load arm assembly). Each attachment subassembly is employedto couple its respective liquid-cooled cold plate 620 to the associatedelectronic component to form the cold plate and electronic componentassemblies. Alignment openings (i.e., thru-holes) are provided on thesides of the cold plate to receive alignment pins or positioning dowelsduring the assembly process. Additionally, connectors (or guide pins)are included within attachment subassembly which facilitate use of theattachment assembly.

As shown in FIG. 6, header subassembly 650 includes two liquidmanifolds, i.e., a coolant supply header 652 and a coolant return header654, which in one embodiment, are coupled together via supportingbrackets. In the monolithic cooling structure of FIG. 6, the coolantsupply header 652 is metallurgically bonded in fluid communication toeach coolant supply tube 640, while the coolant return header 654 ismetallurgically bonded in fluid communication to each coolant returntube 642. A single coolant inlet 651 and a single coolant outlet 653extend from the header subassembly for coupling to the electronicsrack's coolant supply and return manifolds (not shown).

FIG. 6 also depicts one embodiment of the pre-configured,coolant-carrying tubes. In addition to coolant supply tubes 640 andcoolant return tubes 642, bridge tubes or lines 641 are provided forcoupling, for example, a liquid coolant outlet of one liquid-cooled coldplate to the liquid coolant inlet of another liquid-cooled cold plate toconnect in series fluid flow the cold plates, with the pair of coldplates receiving and returning liquid coolant via a respective set ofcoolant supply and return tubes. In one embodiment, the coolant supplytubes 640, bridge tubes 641 and coolant return tubes 642 are eachpre-configured, semi-rigid tubes formed of a thermally conductivematerial, such as copper or aluminum, and the tubes are respectivelybrazed, soldered or welded in a fluid-tight manner to the headersubassembly and/or the liquid-cooled cold plates. The tubes arepre-configured for a particular electronics system to facilitateinstallation of the monolithic structure in engaging relation with theelectronics system.

The above-described liquid cooling approach of FIGS. 2-6 is an effectivesolution for circulating water through liquid-cooled cold platesattached to heat-generating circuit modules (or components). An exampleof the efficacy of this approach is the IBM Power 575™ system offered byInternational Business Machines Corporation, Armonk, N.Y. In theembodiment of FIGS. 2-6, one or more coolant conditioning unitscontaining a pump and, for example, a water-to-water heat exchanger, aredisposed within each electronics rack. As explained above, heat loadcarried by the system coolant circulating through the liquid-cooledcomponents in the liquid-cooled electronics rack is rejected to facilitychilled water passing through the second coolant path through the activewater-to-water heat exchangers within the coolant conditioning unitsdisposed within the rack units.

FIGS. 7-8F depict another aspect of the present invention. Currenthigh-performance computing systems can consume, for example, 250 kW ofenergy per electronics rack. This energy is dissipated in heat, whichposes significant challenges to both the rack-level power and coolingsystems. As noted, to achieve high-performance and high-density,manufacturers use liquid cooling to enable significant performance andcomponent density increases. With the increasing demands for electricalpower and cooling architecture, however, more of the availableelectronics rack volume is needed to house the power and coolingcomponents, limiting the space available for the electronic subsystems(e.g., the computing elements, such as server units). Further, componentredundancy, which ensures availability and reliability of the overallsystem, dictates that multiple (i.e., redundant) power and coolingdevices exist within each electronics rack to service the computingelements in the event of a primary system failure.

Responsive to these issues, disclosed herein is a combined, power andcooling apparatus or rack which includes a frame, at least one bulkpower assembly associated with the frame, and at least one heat exchangeassembly associated with the frame. The bulk power assembly(ies) isconfigured to provide power to one or more electronics racks, whereinthe power and cooling rack is distinct (e.g., separate) from theelectronics rack(s) to be powered and cooled. The one or more heatexchange assemblies are configured to cool system coolant provided tothe electronics rack(s), wherein heat is transferred by the at least oneheat exchange assembly from the system coolant to a facility coolant.Thus, the combined power and cooling apparatus (or rack) provides bothpower and cooling to one or more distinct electronics racks. Byoffloading the power and cooling componentry to a dedicated, combinedpower and cooling rack, space is made available within the one or moreelectronics racks powered and cooled by the combined power and coolingrack to accommodate additional electronic subsystems, such as additionalserver units.

More particularly, disclosed herein in one aspect is a modular,redundant, scalable power and cooling supply rack, designed to servicetwo or more electronics racks of up to 500 kW each. Raw AC or DCelectricity and facility coolant (e.g., building chilled water) aresupplied to the power and cooling rack, which in turn supplies the bulkpower to the computing device inputs of the electronics racks at, forexample, 350 V DC, and cooled system coolant to the electronics racks aswell. Redundancy (n+1) is achieved for two or more electronics racks,rather than for a single electronics rack, thereby advantageouslyreducing the required number of redundant components associated with thepower and cooling of the electronics racks.

FIG. 7 depicts one embodiment of a data center, generally denoted 700,comprising two or more electronics racks 701 and a power and coolingrack 702, in accordance with one or more aspects of the presentinvention. In this embodiment, each electronics rack 701 includes aplurality of electronic subsystems 710, which are liquid-cooled via asystem coolant, for example, passing through one or more liquid-cooledcold plates disposed within the electronic subsystems (or server units).In the embodiment illustrated, each electronics rack 701 includes asystem coolant supply manifold 711 and a system coolant return manifold712, which facilitate the distribution of system coolant received via arespective system coolant loop 742, coupled between the electronics rack701 and at least one heat exchange assembly 740 of the combined powerand cooling rack 702. Additionally, combined power and cooling rack 702includes at least one bulk power assembly 730, which (in one embodiment)comprises an AC-to-DC power supply assembly, coupled to an AC powersupply line 731 and configured to provide conditioned, DC power viadistribution cables 732 to the electronics racks 701.

In one specific example, a three-phase AC source feeds power via ACpower supply line 731 to the bulk power assembly(ies) 730, whichtransforms the supplied AC power to an appropriate DC power level foroutput via the respective power distribution cables 732 to the pluralityof electronics racks 701, and in particular, to the plurality ofelectronic subsystems within the electronics racks. AC power supply line731 supplies (in one example) three phases for international voltage(s).

The number of electronic subsystems installed in an electronics rack 701is variable and depends, for example, on customer requirements for aparticular system. Further, the number of bulk power units within agiven bulk power assembly 730 of the AC-to-DC power supply assembly isalso variable; and is determined (in one implementation) by the numberof electronic subsystems installed in the electronics racks to bepowered by the combined power and cooling rack. More particularly, thenumber of bulk power units (including regulators) within a given bulkpower assembly can be determined by the power requirements of the commonload of the electronics rack(s) fed by the particular bulk powerassembly.

Advantageously, data center 700 depicted in FIG. 7 has a larger volumeavailable within each electronics rack 701 for compute elements, such asserver units, and a smaller overall footprint volume due to theconsolidation of the power and cooling components for multipleelectronics racks within a common rack. For example, the coolantconditioning units, such as described above in connection with FIGS.2-4, employ at the frame level one heat exchanger percoolant-conditioning unit, and there would typically be twocoolant-conditioning units per electronics rack. This is contrasted withthe data center depicted in FIG. 7, wherein (in one embodiment) the heatexchange assembly comprises a single, large heat exchanger.

FIG. 8A depicts a more detailed embodiment of a power and cooling rack702. In this embodiment, the bulk power assembly 730 is shown tocomprise four bulk power units 800, and the heat exchange assembly 740is shown to comprise a single heat exchanger 810, and a plurality ofmodular cooling units (MCUs) 820. By way of example, each bulk powerunit 800 may comprise one or more bulk power regulators as needed toprovide the desired power to one or more electronics racks powered bythe combined power and cooling apparatus.

As described below with reference to FIG. 8D, heat exchanger 810includes a liquid-to-liquid heat exchanger 850 coupled to facilitycoolant loop 741 via a facility coolant supply manifold 830 and afacility coolant return manifold 831. Additionally, the one or moresystem coolant loops 742 are coupled in fluid communication with theheat exchanger 810 via the one or more modular cooling units 820. Inthis embodiment, eight modular cooling units are depicted, by way ofexample only. In one embodiment, four modular cooling units may bededicated to a particular electronics rack of the plurality ofelectronics racks serviced by the combined power and cooling rack 702 toprovide redundancy to the system coolant cooling. For example, twomodular cooling units of four modular cooling units providing systemcoolant to a system coolant manifold (see FIG. 7) associated with aparticular electronics rack may be active at one time, with theremaining modular cooling units offline being as backup to the primarymodular cooling units.

Note with reference to the embodiment of FIG. 8A, that the bulk powerassembly 730 also provides conditioned power 801 to the modular coolingunits 820, and that the facility coolant supply manifold 830 andfacility coolant return manifold 831 provide facility coolant inparallel to the heat exchanger 810 (via a facility coolant inlet line832 and outlet line 833), as well as to, for example, one or more coldplates associated with the bulk power assembly 730 via respectivefacility coolant sub-loops, each comprising a facility coolant supplyline 834 and a facility coolant return line 835. As explained furtherbelow, the facility coolant advantageously directly cools one or morecomponents of the bulk power assembly(ies), which ensures that coolingcapability is available even when the combined power and cooling rack isin standby mode (i.e., a low-power, energy conservation mode) and, forexample, the modular cooling unit's system coolant pumps aredeactivated. As noted, the bulk power assembly(ies) provide (in oneembodiment) conditioned, DC power to the electronics racks, as well asto the modular cooling units and the heat exchange assembly. The heatexchange assembly cools the system coolant, which is then pumped by themodular cooling units to the electronics racks being serviced by thecombined power and cooling rack 702 for use at the electronics racks tocool one or more electronic components thereof, for example, asdescribed above in connection with FIGS. 2-6.

FIG. 8B depicts one embodiment of a modular cooling unit 820, inaccordance with an aspect of the present invention. In this embodiment,modular cooling unit 820 includes a system coolant pump 821, a systemcoolant reservoir 822, and a coolant leak sensor 823 located, forexample, in a lower portion of modular cooling unit 820. A controller827 associated with modular cooling unit 820, or disposed separatelyfrom the modular cooling unit (for example, in the bulk power assembly)senses, for example, system coolant level within system coolantreservoir 822, as well as monitors leak detector 823. The controller 827is coupled to system coolant pump 821 to, for example, controlactivation or deactivation of the pump. As illustrated, quick connectcouplings 824 facilitate connecting modular cooling unit 820 to therespective system coolant loop 742. System coolant loop 742 couples themodular cooling unit to the respective electronics rack, that is, to therack-level system coolant supply and return manifolds described abovewith reference to, for example, FIGS. 2 & 7. The modular cooling unit820 further includes quick connect couplings 825 coupling the coolingunit to, for example, the liquid-to-liquid heat exchanger of the heatexchange assembly 810 (see FIG. 8D). By way of example, the quickconnect couplings referenced herein may comprise various types ofcommercially available couplings, such as those available from ColderProducts Company, of St. Paul, Minn., U.S.A., or Parker Hannifin, ofCleveland, Ohio, U.S.A.

In operation, controller 827 monitors the system coolant reservoir levelsensor(s) and leak sensor(s), interprets the sensors, and controlsactivation or deactivation of system coolant pump 821, and/or controlsmotor speed of the system coolant pump 821, for example, dependent on alevel of system coolant within system coolant reservoir 822.

FIG. 8C depicts one embodiment of a control process which may beimplemented by controller 827. In this process, the controller monitorssystem coolant leak and level sensors for a potential fault 836, anddetermines whether a fault is detected 837. If “no”, processing waitstime “t” 838 before again checking the leak and level sensors for apotential fault 836. Responsive to a fault being detected, thecontroller deactivates the system coolant pump, and initiates a call forservice 839.

FIG. 8D depicts one embodiment of heat exchanger 810. As illustrated,heat exchanger 810 includes a liquid-to-liquid heat exchanger 850, witha system coolant path 851 and a facility coolant path 852 through theliquid-to-liquid heat exchanger 850. System coolant path 851 is coupled(via respective quick connects 825) in fluid communication with theaggregated flow of system coolant through the MCUs and the systemcoolant loops (see FIG. 8A), and facility coolant path 852 is coupled influid communication with facility coolant supply line 832 and facilitycoolant return line 833 via respective quick connects 855. Asillustrated, a controller 857 is coupled to one or more thermal sensors856 for sensing, for example, temperature of system coolant exiting theheat exchange assembly, as well as to (in the depicted embodiment)redundant control valves 858 in fluid communication with facilitycoolant inlet line 832 to the heat exchanger 810. Controller 857 andcontroller 827 (see FIG. 8B) of the modular cooling units may be thesame controller or different controllers. Further, controller 857 couldbe implemented within heat exchange assembly 810 or, for example,remotely within (for example) the combined power and cooling rack, suchas within the bulk power assembly(ies). Each of the redundant controlvalves 858 may be a fail-closed valve, and in one embodiment, only onecontrol valve 858 is open at a time to allow the flow of facilitycoolant therethrough.

As one specific operational example, liquid-to-liquid heat exchanger 850receives facility coolant metered by one or more control valves 858. Thefacility coolant passes through one side of the liquid-to-liquid heatexchanger, and then returns to the facility coolant loop of the datacenter. The control valve 858 is configured (in one embodiment) to closeunder spring force, such that a failure of the control valve (or itspower or control function) results in a closed valve. The secondredundant control valve is in parallel with the first control valve toensure reliable operation in the event of a valve failure. The otherside of the liquid-to-liquid heat exchanger receives system coolant fromthe multiple modular cooling units. A set of redundant temperaturesensors 856 is employed for monitoring temperature of the system coolantbeing supplied to the electronics racks, which may be used, for example,to adjust flow of facility coolant through the liquid-to-liquid heatexchanger via control valve(s) 858, as explained further below withreference to FIG. 8E.

FIG. 8E depicts one process embodiment for controlling facility coolantflow through the liquid-to-liquid heat exchanger responsive to monitoredtemperature of system coolant being output (via the MCUs) to the systemcoolant loops coupled to the electronics racks. A set point temperatureT is obtained, and one of the redundant valves 858 is selected 880. Thevalve is opened by, for example, x steps 881, determined, for example,with reference to the selected set point temperature and (for example)the anticipated temperature of facility coolant provided to the heatexchange assembly. Processing determines via valve feedback whether thevalve is at the selected setting 882. If “no”, a valve fault is posted,and the controller calls for servicing of the valve. The other valve isselected 883, and processing then opens the other valve at the selectedx steps. Assuming that the valve has been opened by the desired amount,then processing obtains system coolant output temperature (T_(SC)) 884,and determines an appropriate valve setting based on the sensedtemperature (T_(SC)) 885. For example, a feedback control mechanism,such as proportional integral differential processing, could be employedat this step to determine the desired valve setting based on the sensedsystem coolant temperature. If appropriate, the valve setting isadjusted. Processing then waits a time “t” 886, before repeating theprocess.

FIG. 8F depicts one embodiment of bulk power assembly cooling, inaccordance with an aspect of the present invention. As illustrated, inthe depicted embodiment, a facility-cooled cold plate 860 is disposed inthermal contact with (e.g., over) one or more electronic components 870of a bulk power unit of the bulk power assembly. A facility coolantsub-loop 861 (e.g., comprising one or more coolant-carrying channelsextending through the cold plate) is associated with the facility-cooledcold plate 860 and is therefore in thermal contact with the electroniccomponents 870 of the respective bulk power unit of the bulk powerassembly. Quick connect couplings 865 may be employed coupling thefacility coolant sub-loop 861 to the respective facility coolant inletline 834 and facility coolant outlet line 835, which as noted above, arein fluid communication with the facility coolant supply manifold andfacility coolant return manifold of the combined power and cooling rack.

One or more thermostats 862 are provided in series fluid communicationwith the facility coolant sub-loop 861 and in thermal contact with oneor more respective heat-generating electronic components 871 of the bulkpower unit being cooled. As component temperature increases, thethermostat valve is configured and disposed to automatically openproportionately, allowing increased facility coolant flow through thefacility coolant sub-loop 861. (By way of example, the one or morethermostats 862 may comprise any of a variety of commercially availablethermostats, such as a Danfoss FJVA series reverse acting thermostaticvalve, offered by Danfoss of Baltimore, Md., USA.) The facility coolantflows through the facility coolant sub-loop, including one or morepassages in the liquid-cooled cold plate 860, absorbing heat from thecomponents, and carrying the heat out to the facility coolant returnmanifold, and hence, to the facility coolant loop.

In operation, the bulk power units of the bulk power assembly supplyconditioned power to one or more electronics racks of the data center,as well as to the MCUs of the heat exchange assembly. In a minimumpower, standby state, the MCUs do not pump system coolant, since theelectronics racks do not require cooling in this state, as no (orminimal) computing work is being performed, and the bulk power units areconverting only enough electrical power to maintain networkcommunications, which can be used to wake the system. In standby state,the only cooling necessary for the power and cooling rack is in the bulkpower units of the bulk power assembly. This cooling is provided by thefacility coolant directly (i.e., without use of system coolant), sincethe one or more thermostats associated with the facility coolantsub-loop(s) may operate independently, allowing, for example, thethermostat valve to open as the associated electronic componenttemperature increases.

As will be appreciated by one skilled in the art, control aspects of thepresent invention may be embodied as a system, method or computerprogram product. Accordingly, control aspects of the present inventionmay take the form of an entirely hardware embodiment, an entirelysoftware embodiment (including firmware, resident software, micro-code,etc.) or an embodiment combining software and hardware aspects that mayall generally be referred to herein as a “circuit,” “module” or“system”. Furthermore, control aspects of the present invention may takethe form of a computer program product embodied in one or more computerreadable medium(s) having computer readable program code embodiedthereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readablestorage medium. A computer readable storage medium may be, for example,but not limited to, an electronic, magnetic, optical, electromagnetic,infrared or semiconductor system, apparatus, or device, or any suitablecombination of the foregoing. More specific examples (a non-exhaustivelist) of the computer readable storage medium include the following: anelectrical connection having one or more wires, a portable computerdiskette, a hard disk, a random access memory (RAM), a read-only memory(ROM), an erasable programmable read-only memory (EPROM or Flashmemory), an optical fiber, a portable compact disc read-only memory(CD-ROM), an optical storage device, a magnetic storage device, or anysuitable combination of the foregoing. In the context of this document,a computer readable storage medium may be any tangible, non-transitorymedium that can contain or store a program for use by or in connectionwith an instruction execution system, apparatus, or device.

In one example, a computer program product includes, for instance, oneor more computer readable storage media to store computer readableprogram code means or logic thereon to provide and facilitate one ormore aspects of the present invention.

Program code embodied on a computer readable medium may be transmittedusing an appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programminglanguage, such as Java, Smalltalk, C++ or the like, and conventionalprocedural programming languages, such as the “C” programming language,assembler or similar programming languages.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprise” (andany form of comprise, such as “comprises” and “comprising”), “have” (andany form of have, such as “has” and “having”), “include” (and any formof include, such as “includes” and “including”), and “contain” (and anyform contain, such as “contains” and “containing”) are open-endedlinking verbs. As a result, a method or device that “comprises”, “has”,“includes” or “contains” one or more steps or elements possesses thoseone or more steps or elements, but is not limited to possessing onlythose one or more steps or elements. Likewise, a step of a method or anelement of a device that “comprises”, “has”, “includes” or “contains”one or more features possesses those one or more features, but is notlimited to possessing only those one or more features. Furthermore, adevice or structure that is configured in a certain way is configured inat least that way, but may also be configured in ways that are notlisted.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below, if any, areintended to include any structure, material, or act for performing thefunction in combination with other claimed elements as specificallyclaimed. The description of the present invention has been presented forpurposes of illustration and description, but is not intended to beexhaustive or limited to the invention in the form disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the invention.

What is claimed is:
 1. An apparatus comprising: a frame; at least onebulk power assembly associated with the frame, and configured to providepower to at least one electronics rack, wherein the apparatus isdistinct from the at least one electronics rack; at least one heatexchange assembly associated with the frame and configured to coolsystem coolant provided to the at least one electronics rack, whereinheat is transferred by the at least one heat exchange assembly from thesystem coolant to a facility coolant, and wherein the apparatus providesboth power and cooling to the at least one electronics rack; and atleast one facility coolant sub-loop and at least one facilitycoolant-cooled cold plate, the at least one facility coolant sub-loopbeing coupled in fluid connection with the at least one facilitycoolant-cooled cold plate, and the at least one facility coolant-cooledcold plate is disposed in thermal conductive contact with at least oneheat-generating component of the at least one bulk power assembly, theat least one facility coolant sub-loop and at least one facilitycoolant-cooled cold plate facilitating transfer of heat from the atleast one heat-generating component of the at least one bulk powerassembly directly to the facility coolant.
 2. The apparatus of claim 1,wherein the at least one bulk power assembly is configured to providepower to multiple electronics racks, the apparatus being distinct fromthe multiple electronics racks, and the at least one heat exchangeassembly is configured to cool system coolant provided to the multipleelectronics racks by transferring heat from the system coolant to thefacility coolant.
 3. The apparatus of claim 1, wherein the at least oneheat exchange assembly comprises at least one liquid-to-liquid heatexchanger across which heat is transferred from the system coolant tothe facility coolant.
 4. The apparatus of claim 3, further comprising atleast one adjustable valve in fluid communication with the at least oneliquid-to-liquid heat exchanger for controlling flow of facility coolantthrough the at least one liquid-to-liquid heat exchanger, and acontroller controlling the at least one adjustable valve based on asensed temperature of the system coolant.
 5. The apparatus of claim 3,further comprising at least one modular cooling unit coupled in fluidcommunication with at least one system coolant loop, the at least onesystem coolant loop facilitating transfer of system coolant between theat least one liquid-to-liquid heat exchanger and the at least oneelectronics rack, each modular cooling unit of the at least one modularcooling unit comprising a system coolant pump and a system coolantreservoir, the system coolant pump facilitating circulation of systemcoolant within the at least one system coolant loop between the at leastone liquid-to-liquid heat exchanger and the at least one electronicsrack.
 6. The apparatus of claim 5, wherein the at least one bulk powerassembly is electrically connected to power the at least one systemcoolant pump of the at least one modular cooling unit.
 7. The apparatusof claim 5, further comprising a controller controlling the at least onesystem coolant pump based, at least in part, on a level of systemcoolant in the system coolant reservoir.
 8. The apparatus of claim 1,further comprising at least one thermostat in thermal contact with theat least one heat-generating component of the at least one bulk powerassembly, the at least one thermostat being configured and disposed toautomatically adjust facility coolant flow through the at least onefacility coolant sub-loop based on a temperature of the at least oneheat-generating component.
 9. The apparatus of claim 8, furthercomprising at least one cold plate coupled to the at least oneheat-generating component of the at least one bulk power assembly, theat least one facility coolant sub-loop being in thermal contact with theat least one cold plate.
 10. The apparatus of claim 8, furthercomprising a facility coolant supply manifold and a facility coolantreturn manifold coupled to facilitate flow of facility coolant inparallel to the at least one heat exchange assembly and the at least onefacility coolant sub-loop associated with the at least one bulk powerassembly.
 11. A data center comprising: at least one electronics rackcomprising at least one electronic subsystem to be powered and cooled;and a power and cooling apparatus for powering and cooling the at leastone electronics rack, the power and cooling apparatus being distinctfrom the at least one electronics rack, and comprising: a frame; atleast one bulk power assembly associated with the frame, and configuredto provide power to the at least one electronics rack; at least one heatexchange assembly associated with the frame and configured to coolsystem coolant provided to the at least one electronics rack, whereinheat is transferred by the at least one heat exchange assembly from thesystem coolant to a facility coolant, and wherein the power and coolingapparatus provides both power and cooling to the at least oneelectronics rack; and at least one facility coolant sub-loop and atleast one facility coolant-cooled cold plate, the at least one facilitycoolant sub-loop being coupled in fluid connection with the at least onefacility coolant-cooled cold plate, and the at least one facilitycoolant-cooled cold plate is disposed in thermal conductive contact withat least one heat-generating component of the at least one bulk powerassembly, the at least one facility coolant sub-loop and at least onefacility coolant-cooled cold plate facilitating transfer of heat fromthe at least one heat-generating component of the at least one bulkpower assembly directly to the facility coolant.
 12. The data center ofclaim 11, wherein the data center comprises multiple electronics racks,and wherein the at least one bulk power assembly is configured toprovide power to the multiple electronics racks, and the at least oneheat exchange assembly is configured to cool system coolant provided tothe multiple electronics racks by transferring heat from the systemcoolant to the facility coolant.
 13. The data center of claim 11,wherein at least one heat exchange assembly comprises at least oneliquid-to-liquid heat exchanger across which heat is transferred fromthe system coolant to the facility coolant, and wherein the power andcooling apparatus further comprises at least one adjustable valve influid communication with the at least one liquid-to-liquid heatexchanger for controlling flow of facility coolant through the at leastone liquid-to-liquid heat exchanger, and a controller controlling the atleast one adjustable valve based on a sensed temperature of the systemcoolant.
 14. The data center of claim 11, wherein the at least one heatexchange assembly comprises at least one liquid-to-liquid heat exchangeracross which heat is transferred from the system coolant to the facilitycoolant, and wherein the power and cooling apparatus further comprisesat least one modular cooling unit coupled in fluid communication with atleast one system coolant loop, the at least one system coolant loopfacilitating transfer of system coolant between the at least oneliquid-to-liquid heat exchanger and the at least one electronics rack,each modular cooling unit of the at least one modular cooling unitcomprising a system coolant pump and a system coolant reservoir, thesystem coolant pump facilitating circulation of system coolant withinthe at least one system coolant loop between the at least oneliquid-to-liquid heat exchanger and the at least one electronics rack.15. The data center of claim 14, wherein the at least one bulk powerassembly is electrically connected to power the at least one systemcoolant pump of the at least one modular cooling unit.
 16. The datacenter of claim 11, wherein the power and cooling apparatus furthercomprises at least one thermostat in thermal contact with at least oneheat-generating component of the at least one bulk power assembly, theat least one thermostat being configured and disposed to automaticallyadjust facility coolant flow through the at least one facility coolantsub-loop based on a temperature of the at least one heat-generatingcomponent.
 17. The data center of claim 16, wherein the power andcooling apparatus further comprises at least one cold plate coupled tothe at least one heat-generating component of the at least one hulkpower assembly, the at least one facility coolant sub-loop being inthermal contact with the at least one cold plate.
 18. The data center ofclaim 17, wherein the power and cooling apparatus further comprises afacility coolant supply manifold and a facility coolant return manifoldcoupled to facilitate flow of facility coolant in parallel to the atleast one heat exchange assembly and the at least one bulk powerassembly.